An investigation of the state formation and transition limitations for prediction problems in recurrent neural networks

Recurrent neural networks are able to store information about previous as well as current inputs. This “memory” allows them to solve temporal problems such as language recognition and sequence prediction, and provide memory elements for larger cognitive networks. It is generally understood that there is an (increasing) relationship between the number of nodes (and connections) in a network, the capabilities of the network, and the amount of training required. However the specifics of this relationship are less well understood. In particular, given that the state of a recurrent network is encoded as a real-valued vector of activation levels, even for small networks there are infinitely many states to choose from. What then determines, or limits, the capabilities of the network? In this paper we use dynamical systems techniques to examine this question in regard to temporal lag. We show that for simple delay problems that the network is unable to solve, the system is able to learn sufficient state representations, but appears to be unable to create transitions that allow it to access those states in the correct order (or equivalently, is unable to arrange its states to suit the transitions that it can support).